Structure of Stereoscopic Image Data, Stereoscopic Image Data Recording Method, Reproducing Method, Recording Program, and Reproducing Program

ABSTRACT

It is made possible to record stereoscopic image data of parallel-ray one-dimensional IP type in a format at a high compression rate with little image quality degradation. This stereoscopic image data can be efficiently decompressed and reproduced. A stereoscopic image data structure includes: a parallax component image data representing n or more parallax component images, each having accumulated pixels that cause the pixels to generate the parallel light rays in the same parallax direction in the viewing zone, and having different numbers of horizontal pixels. N combined images with the same numbers of vertical and horizontal pixels are a unit to be converted into a parallax interleaved image, the n combined images being formed by combining two or more parallax component images with parallax directions different from each other by n.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based upon and claims the benefit of priority fromprior Japanese Patent Application No. 2005-251412 filed on Aug. 31, 2005in Japan, the entire contents of which are incorporated herein byreference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a stereoscopic image data structure, astereoscopic image data recording method, a reproducing method, arecording program, and a reproducing program.

2. Related Art

There have been various types of stereoscopic display devices orthree-dimensional display devices that can display moving stereoscopicimages. In recent years, there has been an increasing demand forstereoscopic display devices of flat panel types that do not requirespecial glasses or the likes. In a flat-panel display device such as aliquid crystal display device or a plasma display device of a directview type or a projection type, the pixel locations in the display planeare fixed, and a parallax barrier that controls the light rays from thedisplay panel toward the viewer is provided directly in front of thedisplay panel. Thus, a stereoscopic display device can be produced withrelative ease.

Through a parallax barrier, light rays are controlled in such a mannerthat different images can be seen from different angles even when thesame location on the parallax barrier is viewed. More specifically, in acase where only right and left parallaxes (horizontal disparities) aregiven, a slit sheet or a lenticular sheet (a cylindrical lens array) isused. In a case where vertical disparities are also given, a pinholearray or a lens array is used. Structures with parallax barriers arefurther classified into binocular types, multiview types,super-multiview types (multiview types under super-multiviewconditions), and integral photography (IP) types. The principles ofthose structures are basically the same as the principles ofstereoscopic photography invented almost 100 years ago.

Generally, in a structure of the IP type or the multiview type, theviewing distance is limited, and therefore, a display image is createdso that a perspectively projected image can be actually viewed at theviewing distance. In the structure of the IP type only with thehorizontal disparities (the one-dimensional IP type, see “SID04 Digest1438” (2004), for example), sets of parallel light rays are formed in acase where the horizontal pitch of the parallax barrier is set at anintegral multiple (n) of the horizontal pitch of pixels (this IP typewill be hereinafter also referred to as the “parallel-rayone-dimensional IP type”). Accordingly, a parallax component image inwhich pixel columns forming sets of parallel light rays are integratedis a perspectively projected image with a predetermined viewing distancein the vertical direction while being an orthographically projectedimage in the horizontal direction. Each parallax component image that isa perspectively projected image in the vertical direction while being anorthographically projected image in the horizontal direction is dividedinto pixel columns, and the pixel columns are rearranged in aninterleaving manner, so as to form a parallax interleaved image (anelemental image array). The parallax interleaved image is displayed onthe display plane and is viewed through the parallax barrier. In thismanner, a stereoscopic image is obtained through normal projection,which is perspective projection in both the horizontal direction and thevertical direction. This method is described in greater detail in “SID04Digest 1438” (2004). In a structure of the multiview type, images formedthrough simple perspective projection are divided into pixel columns andare rearranged in an interleaving manner, so as to form a stereoscopicimage with normal projection.

An image-taking device that uses different projecting methods anddifferent projection center distances depending on the direction (thevertical or horizontal direction) is difficult to produce, because acamera or lenses of the same size as each object are required fororthographic projection. Therefore, to obtain orthographic projectiondata through image taking, a method of converting perspective projectiondata into orthographic projection data is preferred in practice. As anexample of such a method, the “ray space method” that involvesinterpolation using an “EPI (epipolar plane)” is known.

The parallel-ray one-dimensional IP is more advantageous in viewabilitythan the binocular method. However, in a structure of the parallel-rayone-dimensional IP type, the image format is complicated in terms ofprojection and division allocation. In a binocular or multiviewstructure that is one of the simplest stereoscopic display structures,the image format is also simple, and the images from all the viewpointsare formed with the same numbers of vertical and horizontal pixels. Twoparallax component images in the case of the binocular type, or nineparallax components images in the case of a nine-lens type, are dividedinto pixel columns, and the pixel columns are rearranged into a parallaxinterleaved image to be displayed on the display plane.

Compared with the multiview type with similar resolutions, the number ofparallax components images is larger in a structure of the parallel-rayone-dimensional IP type, and the numbers of horizontal pixels (or thehorizontal ranges to be used) of the parallax component images vary withthe parallax directions. As a result, the image format is complicated.With these facts being taken into consideration, the inventors suggesteda method of efficiently recording a stereoscopic image involving a highcompression rate with little degradation of image quality (JapanesePatent Application No. 2004-285246).

The cylindrical lenses of a lenticular sheet may extend diagonally,instead of vertically (see JP-A KOKAI No. 2001-501073). The inventorsalso discovered that the parallel-ray one-dimensional IP type could beapplied to a structure of the slanted lens type (Japanese PatentApplication No. 2004-32973).

In a case where parallax information is allocated to each sub-pixel in astructure of the multiview type or the parallel-ray one-dimensional IPtype, the parallax information is mixed when images in the form ofparallax interleaved images are irreversibly compressed by an encodingmethod such as JPEG or MPEG. As a result, the image quality is degradedat the time of decompression. In a case of reversible (lossless)compression, the problem of image quality degradation is not caused, butthe compression rate is much lower than in the case of irreversible(lossy) compression. Also, a method of irreversibly compressing and thendecompressing parallax component images independently of one another iseasily applied in a structure of a multiview type. However, such amethod is not reasonable in a structure of the parallel-rayone-dimensional IP type that involves a large number of parallaxcomponent images with different numbers of horizontal pixels. Especiallyin a case where the lenses extend diagonally with respect to thevertical direction, the data format and processing become morecomplicated, and it is difficult to achieve a high resolution and a highprocessing speed at the same time.

As described above, the conventional method of recording a stereoscopicimage of a parallel-ray one-dimensional IP type has the problems ofimage quality degradation with a high compression rate at the time ofdecompressing.

SUMMARY OF THE INVENTION

The present invention is proposed in consideration of the aforementionedcircumstances, and it is an object of the present invention to provide astereoscopic image data structure, a recording method, a reproducingmethod, a recording program, and a reproducing program that cause littleimage quality degradation in an efficient process at a high compressionrate in a structure of a parallel-ray one-dimensional IP type usinglenses extending diagonally with respect to the vertical direction.

According to a first aspect of the present invention, there is provideda stereoscopic image data structure for a stereoscopic display devicethat displays a stereoscopic image, with parallaxes being given in ahorizontal direction but not given in a vertical direction,

the stereoscopic display device comprising:

a display unit that has a display face on which a parallax interleavedimage for stereoscopic display is displayed, with pixels being arrangedwith a first horizontal pitch in the horizontal direction; and

a parallax barrier that has linear optical apertures disposed to facethe display face and arranged with a second horizontal pitch in thehorizontal direction, the optical apertures inclined from the verticaldirection, the second horizontal pitch being equal to an integralmultiple (n) of the first horizontal pitch, the parallax barrierdirecting light rays emitted from pixels at horizontal intervals of npixels as parallel light rays toward the viewing zone,

the stereoscopic image data structure comprising: a parallax componentimage data representing n or more parallax component images, each havingaccumulated pixels that cause the pixels to generate the parallel lightrays in the same parallax direction in the viewing zone, and havingdifferent numbers of horizontal pixels,

wherein n combined images with the same numbers of vertical andhorizontal pixels are a unit to be converted into a parallax interleavedimage, the n combined images being formed by combining one or moreparallax component images with parallax directions different from eachother by n.

According to a second aspect of the present invention, there is provideda method of recording stereoscopic image data for a stereoscopic displaydevice that displays a stereoscopic image, with parallaxes being givenin a horizontal direction but not given in a vertical direction,

the stereoscopic display device including:

a display unit that has a display face on which a parallax interleavedimage for stereoscopic display is displayed, with pixels being arrangedwith a first horizontal pitch in the horizontal direction; and

a parallax barrier that has linear optical apertures disposed to facethe display face and arranged with a second horizontal pitch in thehorizontal direction, the optical apertures inclined from the verticaldirection, the second horizontal pitch being equal to an integermultiple (n) of the first horizontal pitch, the parallax barrierdirecting light rays emitted from pixels at horizontal intervals of npixels as parallel light rays toward the viewing zone,

the method comprising:

preparing a parallax component image data representing n or moreparallax component images, each having accumulated pixels that cause thepixels to generate the parallel light rays in the same parallaxdirection in the viewing zone, and having different numbers ofhorizontal pixels; and

recording n combined images with the same numbers of vertical andhorizontal pixels as a unit to be converted into a parallax interleavedimage, the n combined images being formed by combining one or moreparallax component images with parallax directions different from eachother by n.

According to a third aspect of the present invention, there is provideda reproducing method for a stereoscopic image, with parallaxes beinggiven in a horizontal direction but not given in a vertical direction,

the stereoscopic display device including:

a display unit that has a display face on which a parallax interleavedimage for stereoscopic display is displayed, with pixels being arrangedwith a first horizontal pitch in the horizontal direction; and

a parallax barrier that has linear optical apertures disposed to facethe display face and arranged with a second horizontal pitch in thehorizontal direction, the optical apertures inclined from the verticaldirection, the second horizontal pitch being equal to an integermultiple (n) of the first horizontal pitch, the parallax barrierdirecting light rays emitted from pixels at horizontal intervals of npixels as parallel light rays toward the viewing zone,

the method comprising:

preparing a parallax component image data representing n or moreparallax component images, each having accumulated pixels that cause thepixels to generate the parallel light rays in the same parallaxdirection in the viewing zone, and having different numbers ofhorizontal pixels;

recording n combined images with the same numbers of vertical andhorizontal pixels, the n combined images being formed by combining oneor more parallax component images with parallax directions differentfrom each other by n; and

displaying a parallax interleaved image on the display face afterconverting the n combined images into the parallax interleaved image.

According to a fourth aspect of the present invention, there is provideda reproducing method for a stereoscopic image, with parallaxes beinggiven in a horizontal direction but not given in a vertical direction,

the stereoscopic display device including:

a display unit that has a display face on which a parallax interleavedimage for stereoscopic display is displayed, with pixels being arrangedwith a first horizontal pitch in the horizontal direction; and

a parallax barrier that has linear optical apertures disposed to facethe display face and arranged with a second horizontal pitch in thehorizontal direction, the optical apertures inclined from the verticaldirection, the second horizontal pitch being equal to an integermultiple (n) of the first horizontal pitch, the parallax barrierdirecting light rays emitted from pixels at horizontal intervals of npixels as parallel light rays toward the viewing zone,

the method comprising:

preparing a parallax component image data representing n or moreparallax component images, each having accumulated pixels that cause thepixels to generate the parallel light rays in the same parallaxdirection in the viewing zone, and having different numbers ofhorizontal pixels;

recording an ultimate combined image that is formed by combining ncombined images having the same numbers of vertical and horizontalpixels, the n combined images being formed by combining one or moreparallax component images with parallax directions different from eachother by n; and

displaying a parallax interleaved image on the display face afterconverting the ultimate combined image into the parallax interleavedimage.

According to a fifth aspect of the present invention, there is provideda computer-executable program for recording stereoscopic image data fora stereoscopic display device that displays a stereoscopic image, withparallaxes being given in a horizontal direction but not given in avertical direction,

the program comprising instructions for:

preparing a parallax component image data representing n or moreparallax component images, each having accumulated pixels that cause thepixels to generate the parallel light rays in the same parallaxdirection in the viewing zone, and having different numbers ofhorizontal pixels; and

recording n combined images with the same numbers of vertical andhorizontal pixels, the n combined images being formed by combining oneor more parallax component images with parallax directions differentfrom each other by n.

According to a sixth aspect of the present invention, there is provideda computer-executable reproducing program for displaying a stereoscopicimage, with parallaxes being given in a horizontal direction but notgiven in a vertical direction,

the program comprising instructions for:

preparing a parallax component image data representing n or moreparallax component images, each having accumulated pixels that cause thepixels to generate the parallel light rays in the same parallaxdirection in the viewing zone, and having different numbers ofhorizontal pixels;

recording n combined images with the same numbers of vertical andhorizontal pixels, the n combined images being formed by combining oneor more parallax component images with parallax directions differentfrom each other by n; and

displaying a parallax interleaved image on a display unit afterconverting the n combined images into the parallax interleaved image.

According to a seventh aspect of the present invention, there isprovided a computer-executable reproducing program for displaying astereoscopic image, with parallaxes being given in a horizontaldirection but not given in a vertical direction,

the program comprising instructions for:

preparing a parallax component image data representing n or moreparallax component images, each having accumulated pixels that cause thepixels to generate the parallel light rays in the same parallaxdirection in the viewing zone, and having different numbers ofhorizontal pixels;

recording an ultimate combined image that is formed by combining ncombined images having the same numbers of vertical and horizontalpixels, the n combined images being formed by combining one or moreparallax component images with parallax directions different from eachother by n; and

displaying a parallax interleaved image on a display unit afterconverting the ultimate combined image into the parallax interleavedimage.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic perspective view of an entire stereoscopic displaydevice in which a stereoscopic image recording method and a reproducingmethod according to one embodiment of the present invention are applied;

FIG. 2A is a schematic perspective view of a lenticular sheet thatserves as the parallax barrier shown in FIG. 1;

FIG. 2B is a schematic perspective view of a slit sheet that serves asthe parallax barrier shown in FIG. 1;

FIGS. 3( a), 3(b), and 3(c) schematically illustrate a stereoscopicdisplay device for which the stereoscopic image recording method and thereproducing method according to one embodiment of present invention areapplied;

FIGS. 4( a), 4(b), and 4(c) illustrate a method of forming a parallaxinterleaved image based on parallax component images in a structure of aparallel-ray one-dimensional IP type according to one embodiment of thepresent invention;

FIGS. 5( a), 5(b), and 5(c) schematically illustrate the allotment ofthe parallax component images to the parallax interleaved imageaccording to one embodiment of the present invention;

FIG. 6 is a schematic perspective view of part of a stereoscopic displaydevice in which a stereoscopic image recording method and a reproducingmethod are applied;

FIG. 7 is a schematic enlarged plan view of an example of thearrangement of elemental images and effective pixels on the displaydevice shown in FIG. 6;

FIG. 8 is a plan view of the arrangement of combined images having thesame numbers of vertical and horizontal pixels that is suitable forrecording a stereoscopic image for which a stereoscopic image recordingmethod according to one embodiment of present invention is applied;

FIG. 9 schematically illustrates a method of allotting combined imagesin a parallax interleaved image according to one embodiment of thepresent invention;

FIG. 10 is a schematic plan view of parallax component images to beprocessed by the stereoscopic image recording method according to oneembodiment of the present invention;

FIG. 11 illustrate the data ranges of parallax component images and thelocations of the parallax components images in a parallax interleavedimage formed by the stereoscopic image recording method according to oneembodiment of the present invention;

FIG. 12 is a schematic plan view of the format of an ultimate combinedimage formed by a modification of the stereoscopic image recordingmethod according to one embodiment of the present invention;

FIG. 13 is a schematic plan view of the format of an ultimate combinedimage formed by the stereoscopic image recording method according toanother embodiment of the present invention;

FIG. 14 is a schematic plan view of the format of an ultimate combinedimage formed by the stereoscopic image recording method according tostill another embodiment of the present invention;

FIG. 15 is a schematic plan view of the format of an ultimate combinedimage formed by the stereoscopic image recording method according tofurther still another embodiment of the present invention;

FIG. 16 schematically illustrates a method of recording combined imagesor an ultimate combined image formed by the stereoscopic image recordingmethod through irreversible compression, and a method of reproducing theparallax interleaved image from the combined images or the ultimatecombined image through reading, decompressing, and rearranging accordingto the embodiments described above;

FIG. 17 schematically illustrates a shooting method for obtaining theparallax component images shown in FIG. 10;

FIG. 18 is a perspective view of an example of a computer system thatexecutes a stereoscopic image data recording or reproducing programaccording to one embodiment of the present invention; and

FIG. 19 is a block diagram of a computer system that executes astereoscopic image data recording or reproducing program according toone embodiment of the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

The following is a description of embodiments of the present invention,with reference to the accompanying drawings.

Referring first to FIGS. 1 through 7, an IP display device and a displaymethod are described.

FIG. 1 is a schematic perspective view of a stereoscopic display device.The display device that displays a stereoscopic image illustrated inFIG. 1 is equipped with a flat image display unit 331 that displays aparallax interleaved image (an elemental image array) as a flat image. Aparallax barrier 332 that controls light beams from the flat imagedisplay unit 331 is provided on the front face of the display unit 331.The parallax barrier 332 may be a lenticular sheet 334 illustrated inFIG. 2A or a slit sheet 333 illustrated in FIG. 2B. Here, the lenticularsheet 334 or the slit sheet 333 is referred to as the parallax barrier332. The parallax barrier 332 has optical apertures. Where the parallaxbarrier 332 is the lenticular sheet 334, the optical apertures areequivalent to the cylindrical lenses. Where the parallax barrier 332 isthe slit sheet 333, the optical apertures are equivalent to the slitsformed in the slit sheet 333. The optical apertures of the parallaxbarrier 332 practically restrict the light beams emitted from thedisplay unit 331 onto the viewing zone in which a stereoscopic image isto be observed. The optical apertures are provided for the elementalimages constituting a two-dimensional image displayed on the displayunit 331. Accordingly, the parallax interleaved image displayed on thedisplay unit 331 are formed with the elemental images that are the samein number as the optical apertures of the parallax barrier 332. As aresult, the elemental images are projected into the space in the viewingzone via the optical apertures of the parallax barrier 332, so that astereoscopic image is displayed on the front face of the back face ofthe stereoscopic display device.

The parallax barrier 332 may be provided on the back face side of theflat image display unit 331.

This stereoscopic display device is of a one-dimensional IP type. Whenseen from a viewpoint 343 located on an predetermined viewing distanceL, a stereoscopic image to which a horizontal disparity is applied but avertical disparity is not applied is observed. FIG. 3( a) shows thefront face of the stereoscopic display device, and a control unit thatincludes a driving unit 310, a storage unit 312, and an image processingunit 314. FIG. 3( b) shows the arrangement of the optical systems in thehorizontal plane of the stereoscopic display device, and a line group346 that represents the relationship among the elemental image averagewidth Pe, the second horizontal pitch (the horizontal pitch of theapertures of the parallax barrier) Ps, the viewing distance L, and theviewing zone width W. FIG. 3( c) schematically shows the angle of viewin the vertical plane in the viewing space, with respect to the displayunit 331 of the stereoscopic display device shown in FIG. 3( a).

As shown in FIGS. 1 and 3( b), the stereoscopic display device includesthe flat image display unit 331 that displays a flat image such as aliquid crystal display unit, and the parallax barrier 332 having opticalapertures, as described above. The parallax barrier 332 is formed withthe lenticular sheet 334 or the slit sheet 333 that has the opticalapertures each linearly inclined from the vertical direction andcyclically arranged in a direction deviated from the horizontaldirection, as shown in FIGS. 2A and 2B. In a projection-type displaydevice, this parallax barrier 332 is formed with a curved mirror array,or the like. In this stereoscopic display device, the display unit 331is observed from the eye position 343 via the parallax barrier 332within the range of the horizontal viewing angle 341 and the range ofthe vertical viewing angle 342, so that a stereoscopic image can beobserved in the area of the front side and the back side of the flatimage display unit 331. Here, the number of pixels of the flat imagedisplay unit 331 is 1920 in the transverse (horizontal) direction, andis 1200 in the longitudinal (vertical) direction, if counted by a squareminimum unit. The pixels of each minimum unit contain pixels of red (R),green (G), and blue (B). In this specification, a “pixel” means theminimum unit for controlling luminance independently in one frame on thedisplay face. On the other hand, the sub-pixels of red (R), green (G),and blue (B) are “pixel” in a conventional direct-viewing transmissiveliquid crystal panel.

In FIG. 3( b), the width of each elemental image is determined onlyafter the distance (viewing distance) L between the parallax barrier 332and the viewing distance plane 343, the parallax barrier pitch (thehorizontal pitch of the optical apertures of the parallax barrier 332)Ps, and a parallax barrier gap d are determined. More specifically, theelemental image average pitch Pe is determined by the distances betweenthe viewing point on the viewing distance plane 343 and the projectedpoints of the center points of the apertures projected onto the flatimage display unit 331 along the straight line extending toward the midpoint of each two adjacent apertures (the optical apertures of theparallax barrier 332). Reference numeral 346 indicates the linesconnecting the viewing point and the center points of the apertures, andthe viewing zone width W is determined under the condition that theelemental images do not overlap one another on the display surface ofthe flat image display unit 331. As already mentioned, the elementalimages are equivalent to an interleaved image (part of a parallaxinterleaved image) formed with pixels that generate a bundle of lightrays toward the viewing zone between the parallax barrier 332 and theviewing distance plane 343 via an optical aperture of the parallaxbarrier 332. The elemental images are displayed on the display unit 331,and the displayed image is projected to obtain a stereoscopic image.

The flat image display unit 331 is driven on receipt of a display signalfrom the driving circuit 310, so that the parallax interleaved image isdisplayed on the flat image display unit 331. This driving circuit 310has the storage unit 312 as a peripheral device that compresses acombined image formed with parallax component images (explained later)and stores the compressed combined image as stereoscopic image data.This driving circuit 310 also has the image processing unit 314 as aperipheral device that decompresses the compressed stereoscopic imagedata from the storage unit 312, converts the image data into adecompressed combined image, further converts the decompressed combinedimage into a parallax interleaved image, and thus extracts pixel data.

In the structure of a parallel-ray one-dimensional IP type in which thehorizontal pitch Ps of the apertures is set at an integral multiple ofthe pixel pitch Pp, the average pitch Pe of the elemental images is notan integral multiple of the pixel pitch Pp, but has a fraction. In astructure of a general one-dimensional IP type in which the horizontalpitch Ps of the apertures is not set at an integral multiple of thepixel pitch Pp (not forming parallel light rays), the average pitch Peof elemental images is not an integral multiple of the pixel pitch Ppeither, and normally has a fraction. In a structure of a multiview type,on the other hand, the average pitch Pe of elemental images is set at anintegral multiple of the pixel pitch Pp. In a structure of aone-dimensional IP type, the integral quotient obtained by dividing theaverage pitch Ps of apertures by the pixel pitch Pp is referred to as“the number of parallaxes”.

As illustrated in FIGS. 4( a) through 5(c), each elemental image isformed with pixel columns extracted from the parallax component image426 corresponding to the direction of the corresponding group ofparallel light rays. As is apparent from the drawings, a parallaxinterleaved image 427 for displaying a stereoscopic image is formed witha group of elemental images (also referred to as an elemental imagearray), and is also formed with a large number of parallax componentimages 426 that constitute the elemental images.

FIGS. 4( a), 4(b), and 4(c) illustrate the method of producing aparallax interleaved image based on parallax component images in astructure of a parallel-ray one-dimensional IP type. As shown in FIG. 4(a), an object 421 to be displayed is projected onto a projection plane422 provided actually on the plane on which the parallax barrier 332 ofthe stereoscopic display device is disposed. In a structure of aone-dimensional IP type, an image is projected along projection lines425 that extend toward a projection center line 423 that run in parallelwith the projection plane 422 and locate at the center of the plane ofthe viewing distance L, so that the vertical direction is perspectivelyprojected and the horizontal direction is orthographically projected. Inthis projection, the projection lines 425 do not cross one another inthe horizontal direction, but do cross one another in the vertical lineon the projection center line 423. By this projecting technique, thevertical direction of the object 421 is perspectively projected, and thehorizontal direction of the object 421 is orthographically projected, sothat an object image 424 as shown in FIG. 4( b) is formed on theprojection plane 422. In FIG. 4( a), the object image 424 as shown inFIG. 4( b) corresponds to an image projected in a projecting direction428 denoted by reference numeral 1. In the structure of aone-dimensional IP type, the images 424 of the object 421 projected inseveral directions are required, as shown in FIG. 4( a).

The projected image or the parallax component image 426, which isequivalent to the image of one direction in the image formed byperspectively projecting the vertical direction of the object 421 ontothe projection plane 422 and orthographically projecting the horizontaldirection of the object 421 onto the projection plane 422, is dividedinto pixel columns that extend in the vertical direction, as shown inFIG. 4( b). The pixel columns are then allotted to each elemental imagecorresponding to an optical aperture, and are placed in the parallaxinterleaved image 427. The parallax component images 426 are arranged atintervals that are equivalent to the aperture pitch Ps (the horizontalpitch Ps of the optical apertures), in terms of the length of thedisplay face 427 of the display device. Since the optical apertures arediagonally arranged, the corresponding columns among the parallaxcomponent images 426 are arranged virtually in the vertical direction inthe parallax interleaved image 427, but are arranged diagonally in eachparallax component image 426 so as to match the corresponding opticalapertures.

The necessary resolution of each parallax component image 426 is 1/(thenumber of parallaxes) of the parallax interleaved image 427. So as tomake the vertical resolution equal to the horizontal resolution at thetime of 3D display, the number of parallaxes should preferably be thesquare of an integer m, and each of the horizontal and verticalresolutions of each parallax component image with respect to theparallax interleaved image should preferably be 1/m. FIGS. 5( a), 5(b),and 5(c) illustrate a case where the number of parallaxes is 16. In thiscase, the number of horizontal pixels of the parallax interleaved image427 is 1920, and the number of horizontal pixels of each parallaxcomponent image 426 is 480, which is ¼ of the horizontal pixels 1920 ofthe parallax interleaved image 427. As shown in FIGS. 5( a) and 5(b),the RGB sub-pixels of each parallax component image (a camera image)obtained at the time of shooting are arranged in the transversedirection (or in rows), but the sub-pixel data of the RGB sub-pixels arerearranged in a diagonal direction (a direction virtually equal to thedirection of the optical apertures) in the parallax interleaved image427 and are allotted to the pixels in the diagonal direction. With thisconversion allotment, the horizontal resolution in stereoscopic displayin a structure of a one-dimensional IP type involving only thehorizontal parallaxes can be increased.

The horizontally adjacent pixels (sets of RGB being horizontallyadjacent to one another) of each parallax component image 426 arearranged at intervals of 3m of sub-pixels on the parallax interleavedimage 427. This process is repeated for the other projecting directions428, so that the entire parallax interleaved image as a two-dimensionalimage to be displayed on the display face 427 is completed as shown inFIG. 5( c). Although only the eight directions −4, −3, −2, −1, 1, 2, 3,and 4 are shown as the projecting directions 428 in FIG. 4( a), severaltens of directions may be necessary depending on the viewing distance.In the example with the sixteen parallaxes illustrated in FIGS. 5( a)through 7, twenty-six directions are necessary. For a projected image,which is a parallax component image 426, the number of pixels columnswhich is 1/m, is the largest possible number of pixel strings of theparallax interleaved image 427. However, it is only necessary to createthe image data of the columns in the necessary ranges of the respectiveprojecting directions. The necessary ranges will be described later,with reference to FIG. 10.

The projecting directions 428 shown in FIG. 4( a) correspond to theparallax directions in which the parallax component images 426identified by parallax numbers are viewed. The projecting directions 428are not equiangularly set, but the projection center points (the camerapositions) are set at regular intervals on the viewing distance plane,as will be described later with reference to FIG. 17. More specifically,shooting is performed with a camera moving in parallel (in a fixeddirection) on the projection center line 423 at regular intervals, sothat the projection center points are set at regular intervals.

FIG. 6 is a schematic perspective view of a part of the stereoscopicdisplay device. In the example illustrated in FIG. 6, the lenticularsheet 334 that is formed with a cylindrical lens having opticalapertures extending in a diagonal direction (angle=arctan (¼)) isprovided as the parallax barrier 332 on the front face of the displayface of a flat parallax image display unit such as a liquid crystalpanel. On the display face of the display device, pixels 34 are linearlyarranged in both the horizontal and vertical directions in a matrixfashion, with the horizontal to vertical ratio being 3 to 1. The pixels34 are arranged so that red (R), green (G), and blue (B) repeatedlyappear in this order in each row. This color arrangement is generallycalled a stripe arrangement.

FIG. 7 is an enlarged plan view of an example of the pixel arrangementon the display face shown in FIG. 6. In FIG. 7, numerals −8 to 8allotted in each pixel 34 indicate the parallax numbers identifying theparallax component images described with reference to FIG. 4, and eachtwo adjacent parallax numbers are adjacent to each other in a diagonaldirection. In the arrangement illustrated in FIG. 7, one vertical cycleis formed with sixteen rows, and each four rows represent the 3Dvertical resolution.

On the display screen shown in FIG. 6, each twelve columns and four rowsof pixels 34 constitute one effective pixel 43 (one effective pixel 43is framed by a bold line in FIG. 6). As each effective pixel 43 isformed with forty-eight pixels in the structure of this display unit,stereoscopic display with sixteen parallaxes in the horizontal directioncan be performed, with three pixels of RGB being the minimum unit ofparallax information.

In a structure of a parallel-ray one-dimensional IP type, an integralmultiple of the pixel pitch Pp, for example, twelve-pixel pitch is madeequal to the parallax barrier pitch Ps, and the light rays emitted viathe optical apertures of the parallax barriers 332 form a set ofparallel light rays. In this design, the boundaries between elementalimages appear at slightly longer intervals (at the intervals of 12.016pixels, for example) than the total width of twelve pixels. However,since each effective pixel 43 is defined by a unit of pixels 34, thewidth of each effective pixel 43 is set equal to the total width oftwelve columns (forty-eight pixels) or 12.75 columns (fifty-one pixels),depending on the horizontal position on the display face, as shown inFIG. 7. More specifically, the average value of the elemental imagepitch is larger than the total width of twelve pixels, and thehorizontal pitch of the parallax barrier 332 is set equal to the totalwidth of twelve pixels. The shape of each effective pixel 43 consistingof forty-eight pixels also varies with the horizontal position on thedisplay face. FIG. 7 shows an example of the shape of an elemental image(an effective pixel consisting of forty-eight pixels) in the center ofthe screen in the horizontal direction, and five examples of the shapesof effective pixels (each consisting of forty-eight pixels or fifty-onepixels) outside the center region in the horizontal direction.

Referring now to FIGS. 8 through 17, the structure of image dataobtained by converting a parallax interleaved image displayed on thedisplay unit 331 into a suitable format for compression is described.

FIG. 8 shows n (n=16 in this example) of combined images 2 that have thesame number of pixels in the horizontal and vertical direction, and aresuitable for recording a stereoscopic image. The method of recording astereoscopic image according to the present invention is applied for thecombined images 2. Since n is equal to the number of parallaxes, thenumber of combined images 2 will be hereinafter referred to as thenumber of parallaxes n. Each of the combined images 2 includes oneparallax component image 426 or a combination of several parallaxcomponent images 426 (#+13 to #+1 and #−1 to #−13). The n of combinedimages 2 have such a formatted data structure that can be readilyconverted into one parallax interleaved image 427 to be displayed on theflat image display unit 331. The combined images 2 are then distributedon the display unit 331 according to the divided arrangement of theparallax component images and the divided arrangement method describedwith reference to FIGS. 4( a) through 5(c). Thus, the combined images 2can be converted into a parallax interleaved image.

This conversion method is illustrated in FIG. 9. The image data of theone combined image row including the camera image (#−8) at the rightmostend in the viewing zone (the leftmost end in FIG. 9) is placed at everytwelve pixels from the first column at the leftmost end in the parallaxinterleaved image until the rightmost end, while the RGB pixels arediagonally rearranged. The image data of the one combined image rowincluding the camera image (#−7) at the second rightmost end in theviewing zone is continuously placed next to the pixels already placedand is arranged at every twelve pixels until the rightmost end, whilethe RGB pixels are diagonally rearranged. This conversion issequentially performed. Lastly, the image data of the one combined imagerow including the camera image (#8) at the leftmost end in the viewingzone is continuously placed next to the pixels already placed and isarranged at every twelve pixels until the rightmost end, while the RGBpixels are diagonally rearranged. In FIG. 9, part of the image data runsout of the screen. Each one row of the combined images is contained inthe range of four rows in the parallax interleaved image. The second rowof the combined images is converted virtually in the same manner as theabove, but the placement starting position at the leftmost end in theparallax interleaved image shifts to the right by three pixels. Thus,the placement shown in FIG. 7 is performed. The third row of thecombined images is converted in the same manner as the above, exceptthat the placement starting position at the leftmost end in the parallaxinterleaved image shifts to the left by six pixels. The fourth row ofthe combined images is also converted in the same manner as the above,except that the placement starting position at the leftmost end in theparallax interleaved image shifts to the left by three pixels. As forthe fifth row of the combined images, the placement starting position atthe leftmost end in the parallax interleaved image is the same as theplacement starting position for the first row. In this manner, the sameconversion is cyclically performed on each four rows of the combinedimages (each sixteen rows in the parallax interleaved image), so as tocomplete the placement on the entire area of the parallax interleavedimage. However, a different interpolation might be added to theconverting process for each row of the combined images, as will bedescribed later.

Through the above described conversion, the sixteen combined images canbe processed in the same manner as the processing of 16-view images bythe multiview method, and can be converted into a parallax interleavedimage by exactly the same interleaving process. Here, the combinedimages 2 of the arrangement illustrated in FIG. 8 are recorded on astorage medium, or intra-frame compression is performed on the combinedimages 2 of the arrangement, or inter-frame compression is performed bycorrelating the combined images 2 with other combined images 2 ofanother arrangement. Further, compression may be performed bycorrelating adjacent combined images with one another. By doing so, thecompression rate is increased, but the decompressing load becomeslarger.

The reference numerals (#13 to #1 and #−1 to #−13) in FIG. 8 indicatethe numbers allotted to the parallax component images 426 (the samenumbers as the camera numbers). It should be noted that, when thecombined images are identified hereinafter, the combinations of thenumbers (#13 to #1 and #−1 to #−13) allotted to the parallax componentimages 426 are used. In FIG. 8, for example, the combined image 2located at the upper-left end is identified as the combined image (#−8,#+9), and the combined image 2 that is located on the leftmost and isthe third from the top is identified as the combined image (#+1).

In a structure of a one-dimensional IP type that emits parallel lightrays in the horizontal direction, the parallax barrier 332 (a lenticularsheet) is provided on the front face of the display panel. The parallaxbarrier 332 linearly extends so that the optical apertures (thecylindrical lenses of the lenticular sheet) are arranged with ahorizontal pitch that is equal to an integral multiple, for example, atwelvefold, of the horizontal pitch of the pixels (sub-pixels in thisexample) arranged in the display plane.

In the structure of the one-dimensional IP type, the light rays emittedfrom the pixels that are arranged at intervals of twelve (an integralmultiple) pixels in the horizontal direction of the display face aredirected to the viewing zone, so as to reproduce a stereoscopic image.The number of parallax component images 426 combining the image data ofthe set of pixels constituting the parallel light rays in the sameparallax direction is set to 26, which is larger than 16 (=12×4 (rows)/3(color components)). As shown in FIG. 10, the numbers of pixels in thehorizontal direction (the valid pixel ranges) vary among the parallaxcomponent images 426 (#−13 to #−1 and #+1 to #+13).

FIG. 10 shows the sizes of the valid pixel ranges among the cameraimages including the twenty-six parallax component images 426. In FIG.10, the solid lines indicate the valid pixel ranges of the parallaxcomponent images 426, and the broken lines indicate the camera imagesizes equivalent to the display resolutions at the time of stereoscopicdisplay (or the number of vertical and horizontal pixels correspondingto the projection plane at the time of shooting). The number of verticaland horizontal pixels is set at 480 (in the horizontal direction)×300(in the vertical direction) pixels (not sub-pixels). All the parallaxcomponent images 426 have the same total number of vertical pixels, butvary in the number of horizontal pixels. FIG. 11 shows the specificvalues of the valid pixel ranges. The viewer position (the viewing zone)from which a stereoscopic image can be viewed at a viewing distance isequivalent to the width of the position in which the center sixteencameras among the twenty-six cameras are located. The pixel rangescorresponding light rays that fall in the viewing zone are the validpixel ranges.

The parallax component images 426 that form the combined images 2 shownin FIG. 10 are cropped from images that are formed in the range of thecommon projection plane 422 by the cameras set at an predeterminedviewing distance L from the projection plane 422 (equivalent to theplane which focus on the object 421) as shown in FIG. 17. All thecameras are set in the horizontal direction, and have the commonprojection plane. Therefore, shift lens shooting or cropping afterwide-angle shooting is employed as the shooting method.

In FIG. 17, the shooting positions of the cameras are denoted by thecamera numbers (#1 to #13 and #−1 to #−13) shown in FIG. 10. As shown inFIG. 10, the camera numbers (the parallax numbers) are allotted so thatnumber 0 is omitted in the case of n is an even number, and the positivenumbers and the negative numbers are allotted symmetrically about thecenter of the front face of the display plane 422. When a camera takesan image of the object 421 in the same projection plane while beingmoved at regular intervals on the horizontal shooting standard linelocated at the predetermined viewing distance L from the object 421, animage of a space that contains the object 421 is taken. Since the camera#1 and the camera #−1 are located virtually at the center of thehorizontal shooting standard line, the images (parallel light rays)taken by the camera #1 and the camera #−1 fall in the viewing zone andthe range of all the pixels taken by the camera #1 and the camera #−1are used as the parallax component images 426 denoted by #1 and #−1. Asthe camera number becomes larger or smaller, the range that does notfall in the viewing zone increases among the images projected onto theprojection plane 422. As a result, the valid pixel ranges as theparallax component images 426 decrease, while the invalid pixel rangesnot to be used as the parallax component images 426 increase among theprojected images. For example, the images taken by the camera #8 and thecamera #−8 exhibit virtually the same angle of view, but the images(parallel light rays) that fall in the viewing zone are approximately ½of the entire images. As a result, the parallax component images 426 areformed with approximately ½ of the taken images and the rest of thetaken images become invalid pixel ranges.

FIG. 10 shows the relationship among the actually taken images and theparallax component images 426. As shown in FIG. 10, as the camera numberbecomes larger or smaller, the horizontal pixel range that is valid as aparallax component image 426 cropped from the actually taken imagedecreases, while the invalid pixel range increases. Where the camerasare moved at regular intervals on the horizontal shooting standard line,the valid and invalid ranges of the parallax component images 426 thatexhibit a complementary relationship among them as shown in FIG. 10 areformed between the pixel ranges to be used as the parallax componentimages 426 cropped from the actually taken images and the invalid pixelranges not to be used as the parallax components images 426. Forexample, the valid and invalid ranges of the parallax components image426 are formed in the images taken by the camera #−5. Here, the invalidrange is equal to the valid range of the parallax component image 426 ofthe image taken by the camera #12. Accordingly, the combined number ofpixels in the parallax components images 426 of the images taken by thecameras #−5 and #12 is equal to the number of vertical and horizontalpixels in the parallax component image 426 of the image taken by thecamera #1.

The combined images 2 shown in FIG. 8 have the same number of verticaland horizontal pixels, since the parallax component images 426 croppedfrom the actually taken images shown in FIG. 10 are combined. As isapparent from the comparison among the sizes (the numbers of verticaland horizontal pixels) of the parallax component images 426 shown inFIG. 10, the combinations of the parallax component images 426 denotedby parallax numbers that are different from each other by 16 can beconverted into the sixteen combined images 2 having the same numbers ofvertical and horizontal pixels. For example, the combined image 2 at theupper left end in FIG. 8 is equivalent to the combination of theparallax component images 426 denoted by #−8 and #9 that differ by 16 inparallax number. The combined image 2 at the upper right end in FIG. 8is equivalent to the combination of the parallax component images 426denoted by #−5 and #12 that differ by 16 in parallax number. Eachparallax component image 426 cropped from an image taken by a cameralocated outside the viewing zone is combined with a parallax componentimage 426 within the viewing zone. However, the combined portion (thevertical boundary line) is equivalent to the viewing zone edge at thetime of stereoscopic display. For an image with a very small parallax,the parallax component images 426 exhibit relatively high continuousnessat the combined portion. Accordingly, even after irreversibly compressedcombined images are decompressed, the image quality at the combinedportion is hardly degraded. Some of the combined images 2 (the siximages denoted by #−3 through #3 among the sixteen images) each containonly one parallax component image 426. Since all the combined images 2have the same number of vertical and horizontal pixels, the image dataof the combined images 2 can be advantageously processed in the samemanner as the processing of multiview data in a display device of amultiview type.

FIG. 11 is a table showing the specific numbers of horizontal pixels(the ranges of horizontal pixels (not sub-pixels)) in the parallaxcomponent images 426. The numbers shown in FIG. 11 also represent 3Dpixel numbers (lens numbers). This table is created through calculationsbased on the average elemental image width (slightly larger than thetotal width of twelve pixels) determined by the predetermined viewingdistance L. As is apparent from the table shown in FIG. 11, the imagedenoted by the parallax number #−13 (equivalent to the camera number#−13 in FIG. 12) identifying the parallax direction contains only thepixels between the second column and the 30th column among the 480 pixelcolumns in the camera image illustrated in FIG. 10. Accordingly, thesize of the image denoted by #−13 is equivalent to the total width oftwenty-nine pixels. The data of the 29-pixel width is divided at theintervals of twelve pixels, and the RGB pixels that are originallyarranged in the transverse direction are diagonally arranged in apredetermined region in the parallax interleaved image.

Likewise, the image denoted by the parallax number #−11 contains onlythe pixels between the second column and the 123rd column among the 480pixel columns in the camera image illustrated in FIG. 10. Accordingly,the size of the image denoted by #−11 is equivalent to the total widthof 122 pixels. The data of the 122-pixel width is divided at theintervals of twelve pixels, and the RGB pixels are diagonally arrangedin a predetermined region in the parallax interleaved image to bedisplayed on the display unit 331.

One of the combined images 2 shown in FIG. 8 is formed by combining theimage denoted by the parallax number #−13 with the image denoted by theparallax number #4. The total width (the number of horizontal pixels) ofthe combination of the image denoted by the parallax number #−13 and theimage denoted by the parallax number #4 is 29+451=480. Another one ofthe combined images 2 shown in FIG. 8 is formed by combining the imagedenoted by the parallax number #−11 with the image denoted by theparallax number #6. The total width of the combination of the imagedenoted by the parallax number #−13 and the image denoted by theparallax number #4 is 122+358=480. The total width of any othercombination is also 480.

As described above, each parallax component image 426 is an image thatis properly formed, in terms of design, through perspective projectionwith the predetermined viewing distance L or a similar viewing distancein the vertical direction, and orthographic projection in the horizontaldirection. However, perspective projection may be performed both in thevertical direction and the horizontal direction, as long as thedeformation in the stereoscopic image is inconspicuous.

FIG. 12 shows an example in which the sixteen combined images 2 shown inFIG. 8 are further combined linearly so as to form an ultimate combinedimage. This ultimate combined image is formed by connecting each twocombined images 2 of adjacent parallaxes to each other in the horizontaldirection. In this example, the two combined images 2 of the parallaxes(#−8 and #8) located at both further ends among the sixteen parallaxesclose to the front face of the display plane are arranged at both endsof the ultimate combined image. This format is preferred in terms ofhigh-speed conversion and versatility, because the ultimate combinedimage can be converted into a parallax interleaved image in the samemanner as the processing of multiview data in a display device of amultiview type, and the conversion does not depend on the number ofcameras when the predetermined viewing distance is varied.

As shown in FIG. 13, the sixteen combined images 2 having the samenumbers of vertical and horizontal pixels as shown in FIG. 8 may beconnected to one another both in the horizontal direction and thevertical direction, so as to present a tile-like format. This ultimatecombined image in the tile-like format may have the same number ofvertical and horizontal pixels as that in the parallax interleaved imageto be display on the display face at the time of stereoscopic display.With the number of vertical and horizontal pixels in the ultimatecombined image being equal to the number of vertical and horizontalpixels in the parallax interleaved image as the ultimate display image,compressed recording can be performed according to the standard such asMPEG2. More specifically, inter-frame compression or intra-framecompression can be applied, in a case where the ultimate combined imagein the tile-like format shown in FIG. 13 is prepared as a frame, and amoving image that can be stereoscopically viewed is to be reproducedwith many such frames.

The left and right ends of each parallax component image 426 areequivalent to either the ends of the screen or the ends of the viewingzone at the time of stereoscopic display. The connected portions betweenthe parallax component images in the combined images are equivalent tothe ends of the viewing zone, and the connected portions between theconnected images are equivalent to the ends of the screen. In anirreversible compressing process, encoding is performed for eachpredetermined block size, but the connected portions between combinedimages often match the block boundaries.

Although the connected portions between the parallax component images inthe combined images do not often match the block boundaries, degradationof the image quality does not matter, because a stereoscopic imagenaturally splits at the ends of the viewing zone (the boundaries withthe adjacent lobes) and cannot be viewed properly.

Accordingly, even after the ultimate combined image is irreversiblycompressed and is then decompressed, the stereoscopic image can beprotected from adverse influence of the degradation of image quality atthe connected portions.

This format is preferred in terms of high-speed conversion andversatility, because the ultimate combined image can be converted into aparallax interleaved image in the same manner as the processing ofmultiview data in a display device of a multiview type, and theconversion does not depend on the number of cameras when thepredetermined viewing distance is varied.

The conversion from the ultimate combined image to the parallaxinterleaved image in FIG. 13 is the same as the conversion illustratedin FIG. 9. In a case where the optical apertures are not diagonallyarranged but are arranged vertically, the conversion is performedthrough one-to-one mapping between images of the same number of verticaland horizontal pixels. In a case where the optical apertures arediagonally arranged, however, the pixel data of the parallax interleavedimage needs to be generated through an interpolating process based onone or more adjacent pixels in the horizontal direction in the combinedimages, with the fact that the horizontal position of light ray variesamong the pixel rows being taken into consideration. Where thex-coordinate and the y-coordinate of the pixels in the combined imagesprior to conversion are represented by X_(in) and Y_(in), and thex-coordinate and the y-coordinate of the pixels in the convertedparallax interleaved image are represented by X_(out) and Y_(out), thepixel data can be determined by the following linear interpolatingprocess:

k=(2b−3−Y _(in))% b+1

P(X _(out))=(kP(X _(in))+(b−k)P(X _(in)+1))/b

Here, the coordinates X_(in), Y_(in), X_(out), and Y_(out) are integers,the symbol of operation “%” represents the operation for determining aremainder (an integer), the operation “(2b−3−Y_(in)) % b” represents theremainder (an integer) obtained when (2b−3−Y_(in)) is divided by b.Meanwhile, P(X) represents the image data of the pixels of thecoordinate X. As for the coefficient b, the square of b is equal to thenumber of parallaxes. Accordingly, b is four in the case of sixteenparallaxes, and b is five in the case of twenty-five parallaxes. Theabove described linear interpolating process is performed at a highspeed. The generation of the pixel data of the parallax interleavedimage can be performed through a process with a pixel shader.

If the above described conversion is performed via an intermediateformat that collectively has sixteen rows formed with four sets of fourrows extracted from the same locations in the respective stages of thefour-stage structure shown in FIG. 13, mapping is contained within thesixteen rows. Therefore, the use of such an intermediate format might bepreferred in some cases, depending on the processing system to be used.The converting process involving the format and the interpolatingprocess shown in FIG. 13 is suitable for processing an actually shotimage or an existing multi-viewpoint image. In such a case, eachparallax component image is an image on which regular square samplinghas been performed.

As shown in FIG. 14, the combined images in the ultimate combined imagemay be parallelograms, and the image data corresponding to one opticalaperture may be aligned in a vertical line in the combined images.Accordingly, each combined image shows a diagonally deformed picture,but high continuousness can be maintained with respect to the diagonallyextending optical apertures. In this arrangement, inter-framecompression and intra-frame compression can be applied, with littledegradation of the image quality being caused at the time ofirreversible compression recording. Like the conversion illustrated inFIG. 9, the conversion into a parallax interleaved image is performedthrough one-to-one mapping, with each one row in the combined imagescorresponding to four rows in the parallax interleaved image.Accordingly, the interpolating process performed in the exampleillustrated in FIG. 13 becomes unnecessary. However, the cyclicity ofthe converting process for each row in the parallax interleaved image islost. As a result, the processing becomes a little more complicated. Incase of CG (computer graphics) images, vertically dividing rendering canbe performed, with the difference in horizontal position of light rayamong the rows in a parallax interleaved image being taken intoconsideration. Therefore, the format shown in FIG. 14 and the one-to-oneimage transfer are suitable for CG. In such a case, each parallaxcomponent image is an image on which diagonal sampling has beenperformed according to the inclination of the optical apertures. In thecase of an image on which square sampling has been originally performed,such as an actually shot image or an existing multi-viewpoint image, aninterpolating process should be performed at the stage of forming theultimate combined image of FIG. 14.

As shown in FIG. 15, each combined image may have a structure formed bystacking m of parallax component images vertically in m stages. Each ofthe m of parallax component images has a vertical resolution that is 1/mof the horizontal resolution. The parallax component images divided by mare equivalent to the group of rows extracted in vertical cycles(sixteen rows in this example) of the optical apertures. Each of theimages divided by m shows a picture that is 1/m of the original picturein the vertical direction. In this arrangement, inter-frame compressionand intra-frame compression can be applied especially for a large-sizedparallax interleaved image, with little degradation of the image qualitybeing caused at the time of irreversible compression recording. Like theconversion illustrated in FIG. 9, the conversion into a parallaxinterleaved image is performed through one-to-one mapping, with each onerow in the combined images corresponding to four rows in the parallaxinterleaved image. However, the cyclicity of the converting process foreach row in the parallax interleaved image is lost. As a result, theprocessing becomes a little more complicated. In case of CG images,vertically dividing rendering can be performed, with the difference inhorizontal position of light ray among the rows in a parallaxinterleaved image being taken into consideration. Therefore, the formatshown in FIG. 15 and the one-to-one mapping are suitable for CG. In sucha case, each parallax component image is an image on which diagonalsampling has been performed according to the inclination of the opticalapertures. In the case of an image on which square sampling has beenoriginally performed, such as an actually shot image or an existingmulti-viewpoint image, an interpolating process should be performed atthe stage of forming the ultimate combined image of FIG. 15.

The combined images shown in FIG. 8 do not necessarily form an ultimatecombined image in which the combined images are arranged in a plane, butmay be formed in a combined state as a rectangular-parallelepiped rayspace defined by the “ray space method”. In such arectangular-parallelepiped virtual space, compression recording andinterpolating can be performed.

Referring now to FIG. 16, a stereoscopic image data recording and areproducing method are described. FIG. 16 schematically illustrates amethod of recording the combined images 2 or an ultimate combined imageformed by the stereoscopic image recording method through irreversiblecompression, and a method of reproducing the parallax interleaved imagefrom the combined images or the ultimate combined image through reading,decompressing, and rearranging.

As described with reference to FIG. 17, the camera images indicated bythe broken lines in FIG. 10 are first obtained by taking images of theobject 421 to be displayed as a stereoscopic image in the respectivecamera positions (#13 to #1 and #−1 to #−13).

The parallax component images 421 with necessary numbers of vertical andhorizontal pixels indicated by the solid lines in FIG. 10 are extractedfrom the camera images by cutting and resizing process (step S11). Eachone or more parallax components images 421 with parallax numbersdifferent from each other by the number of parallaxes are combined toform a combined image, as shown in FIG. 8, and the combined images arefurther combined and arranged so as to form an ultimate combined image(uncompressed) as shown in FIGS. 12 through 15 (step S12).

In a case where an image is to be generated from CG model data,vertically dividing rendering is performed according to the sets ofparallel light rays emitted in a different horizontal position for eachrow (step S10). The resultant images are then combined and convertedinto an ultimate combined image (uncompressed) (step S12). For thisconverting process, a parallax allocation map that is prepared inadvance may be used. With the cyclicity in the vertical direction beingtaken into consideration, the minimum unit number of rows (sixteen rows,for example) is sufficient for the parallax allocation map. The ultimatecombined image is compressed by an irreversible encoding method with ahigh compression rate, such as JPEG. In a case where the stereoscopicimage to be displayed is a moving image, the ultimate combined image iscorrelated with another ultimate combined image adjacent to the ultimatecombined image in terms of time, and the ultimate combined images arecompressed by an irreversible encoding method with a high compressionrate, such as MPEG.

The compressed ultimate combined image data are stored and saved on astorage medium or the storage unit 312 shown in FIG. 3( a) (Step S13).

At the time of reproduction, the compressed ultimate combined image isdecompressed and converted into the ultimate combined image by the imageprocessing unit 314 shown in FIG. 3( a) (step S14). The pixel row datacorresponding to the optical apertures are extracted from the combinedimages of the ultimate combined image, and are rearranged in a framememory (not shown) with a predetermined pitch, as shown in FIG. 9.

After the ultimate combined image is rearranged in the frame memory, theentire parallax interleaved image is completed as shown in FIG. 4( c)(step S15).

The parallax interleaved image is displayed on the display unit 331, soas to display a stereoscopic image in the viewing zone. In the case ofstreaming via a remote server, the storage unit and the image processingunit are remote from each other.

The uncompressed ultimate combined image may be converted directly intoa parallax interleaved image for display, without a compressing process.This is suitable for a real-time converting process.

In each of the ultimate combined images shown in FIGS. 12 through 15,the locations of all the combined images may be shifted cyclically, sothat the ultimate combined image is converted into a parallaxinterleaved image. For example, the combined image #−2 is shifted to thelocation of the combined image #−1, and the combined image #−1 isshifted to the location of the combined image #1. In this manner, astereoscopic image can be displayed in a shifted viewing zone (shiftedby 1/16 of the viewing zone width). This process is suitable for fineadjustment of the viewing zone or head tracking.

As described above, after the conversion into the combined images 2 withthe same numbers of vertical and horizontal pixels, the combined images2 are compressed. In this manner, adverse influence due to an increaseor a decrease in the number of cameras (parallax directions) or a changein pixel number is prevented when the predetermined viewing distance isvaried. Furthermore, image quality degradation is minimized. Morepreferably, arrangement and combinations are employed so that thecombined images 2 are correlated with one another. Thus, a highercompression rate can be achieved.

The program for executing the stereoscopic image data recording methodand the reproducing method described with reference to FIG. 16 isexecuted by a computer system illustrated in FIGS. 18 and 19.

As shown in FIG. 18, a computer system 130 includes a computer main body131 having a CPU and a GPU, a display device 132 such as a LCD, an inputunit 133 such as a keyboard and a mouse, and a printer 134 that performsprinting.

As shown in FIG. 19, the computer main body 131 has a built-in memory135 formed with a RAM, and a recording (storage) disk drive unit 136that can be provided inside or outside the computer main body 131. Asthe recording disk drive unit 136, a floppy disk (FD) drive 137, anoptical disk drive 138, and a hard disk (HD) drive unit 139 areprovided. As shown in FIG. 18, a floppy disk (FD) 141 to be inserted tothe slot of the FD drive 137, and a CD-ROM, a CD-R, a DVD-RAM, or aDVD-R 142 to be used in the optical disk drive 138 are used as recordingmedia 140 to be used in the recording disk drive unit 136. The recordingmedia 140 may be any other computer-readable media, such as otheroptical recording disks, card memories, and magnetic tapes.

The above described program may be installed in a remote computer thatis connected to a network such as the Internet. In such a case,compressed images are downloaded via the network, and are decompressedand rearranged by a local computer.

The above described program may also be provided or distributed via anetwork such as the Internet.

As described so far, according to the present invention, recording andreproduction with a high compression rate can be efficiently performed,with little degradation in image quality in a structure of aparallel-ray one-dimensional IP type using lenses extending diagonallywith respect to the vertical direction. Like general MPEG data, thestereoscopic image data structure and the recording method according tothe present invention can be employed not only for recording on arecording medium, but also for distribution via a wire or wirelesscommunication means, such as streaming.

The present invention is not limited to the above described embodiments,but various modifications can be made to them in practice withoutdeparting from the scope of the invention.

Also, the components disclosed in the above embodiments may be combinedwith one another in various ways so as to form various other structures.Some of the components may be removed from the components disclosed inthe above embodiments. Furthermore, the components may be combined withcomponents of some other embodiments, if necessary.

Additional advantages and modifications will readily occur to thoseskilled in the art. Therefore, the invention in its broader aspects isnot limited to the specific details and representative embodiments shownand described herein. Accordingly, various modifications may be madewithout departing from the spirit or scope of the general inventiveconcepts as defined by the appended claims and their equivalents.

1. A stereoscopic image data structure for a stereoscopic display devicethat displays a stereoscopic image, with parallaxes being given in ahorizontal direction but not given in a vertical direction, thestereoscopic display device comprising: a display unit that has adisplay face on which a parallax interleaved image for stereoscopicdisplay is displayed, with pixels being arranged with a first horizontalpitch in the horizontal direction; and a parallax barrier that haslinear optical apertures disposed to face the display face and arrangedwith a second horizontal pitch in the horizontal direction, the opticalapertures inclined from the vertical direction, the second horizontalpitch being equal to an integral multiple (n) of the first horizontalpitch, the parallax barrier directing light rays emitted from pixels athorizontal intervals of n pixels as parallel light rays toward theviewing zone, the stereoscopic image data structure comprising: aparallax component image data representing n or more parallax componentimages, each having accumulated pixels that cause the pixels to generatethe parallel light rays in the same parallax direction in the viewingzone, and having different numbers of horizontal pixels, wherein ncombined images with the same numbers of vertical and horizontal pixelsare a unit to be converted into a parallax interleaved image, the ncombined images being formed by combining one or more parallax componentimages with parallax directions different from each other by n.
 2. Amethod of recording stereoscopic image data for a stereoscopic displaydevice that displays a stereoscopic image, with parallaxes being givenin a horizontal direction but not given in a vertical direction, thestereoscopic display device including: a display unit that has a displayface on which a parallax interleaved image for stereoscopic display isdisplayed, with pixels being arranged with a first horizontal pitch inthe horizontal direction; and a parallax barrier that has linear opticalapertures disposed to face the display face and arranged with a secondhorizontal pitch in the horizontal direction, the optical aperturesinclined from the vertical direction, the second horizontal pitch beingequal to an integer multiple (n) of the first horizontal pitch, theparallax barrier directing light rays emitted from pixels at horizontalintervals of n pixels as parallel light rays toward the viewing zone,the method comprising: preparing a parallax component image datarepresenting n or more parallax component images, each havingaccumulated pixels that cause the pixels to generate the parallel lightrays in the same parallax direction in the viewing zone, and havingdifferent numbers of horizontal pixels; and recording n combined imageswith the same numbers of vertical and horizontal pixels as a unit to beconverted into a parallax interleaved image, the n combined images beingformed by combining one or more parallax component images with parallaxdirections different from each other by n.
 3. The method of recordingstereoscopic image data as claimed in claim 2, wherein: each of thecombined images is a parallelogram; and the image data corresponding toone of the linear optical apertures is aligned in a vertical line in thecombined images.
 4. The method of recording stereoscopic image data asclaimed in claim 2, wherein each of the combined images has parallaxcomponent images stacked in m stages, each of the parallax componentimages having a vertical resolution that is 1/m of a horizontalresolution.
 5. The method of recording stereoscopic image data asclaimed in claim 2, wherein each of the parallax component images areformed by perspective projection in vertical direction according to thepredetermined viewing distance and by orthographic projection inhorizontal direction.
 6. The method of recording stereoscopic image dataas claimed in claim 2, wherein each of the parallax component images areformed by perspective projection according to the predetermined viewingdistance.
 7. The method of recording stereoscopic image data as claimedin claim 2, wherein the n combined images are further combined so as toform an ultimate combined image to be recorded.
 8. The method ofrecording stereoscopic image data as claimed in claim 7, wherein theultimate combined image is formed by combining the combined images insuch a manner that the combined images having adjacent parallaxdirections are adjacent to one another in the horizontal direction. 9.The method of recording stereoscopic image data as claimed in claim 8,wherein: the ultimate combined image is formed by combining the combinedimages in such a manner that the combined images having adjacentparallax directions are adjacent to one another in the horizontaldirection; and the two combined images having the parallax directions atboth ends of the n parallax direction close to the front face of thedisplay face are disposed at both ends of the ultimate combined image.10. The method of recording stereoscopic image data as claimed in claim2, wherein the ultimate combined image is formed by combining thecombined images in the horizontal direction and the vertical direction,so as to form a tile-like format.
 11. The method of recordingstereoscopic image data as claimed in claim 2, wherein the ultimatecombined image has the same numbers of vertical and horizontal pixels asthose of the parallax interleaved image displayed on the display face atthe time of stereoscopic display.
 12. The method of recordingstereoscopic image data as claimed in claim 7, wherein the ultimatecombined image is formed as a rectangular-parallelepiped ray spacedefined by a ray space method.
 13. The method of recording stereoscopicimage data as claimed in claim 2, wherein the combined images or theultimate combined image is irreversibly compressed and then recorded.14. A reproducing method for a stereoscopic image, with parallaxes beinggiven in a horizontal direction but not given in a vertical direction,the stereoscopic display device including: a display unit that has adisplay face on which a parallax interleaved image for stereoscopicdisplay is displayed, with pixels being arranged with a first horizontalpitch in the horizontal direction; and a parallax barrier that haslinear optical apertures disposed to face the display face and arrangedwith a second horizontal pitch in the horizontal direction, the opticalapertures inclined from the vertical direction, the second horizontalpitch being equal to an integer multiple (n) of the first horizontalpitch, the parallax barrier directing light rays emitted from pixels athorizontal intervals of n pixels as parallel light rays toward theviewing zone, the method comprising: preparing a parallax componentimage data representing n or more parallax component images, each havingaccumulated pixels that cause the pixels to generate the parallel lightrays in the same parallax direction in the viewing zone, and havingdifferent numbers of horizontal pixels; recording n combined images withthe same numbers of vertical and horizontal pixels, the n combinedimages being formed by combining one or more parallax component imageswith parallax directions different from each other by n; and displayinga parallax interleaved image on the display face after converting the ncombined images into the parallax interleaved image.
 15. The reproducingmethod for a stereoscopic image as claimed in claim 14, wherein: each ofthe combined images is a rectangular having the same aspect ratio as theparallax interleaved image; and the conversion into the parallaxinterleaved image involves generation of pixel data of the parallaxinterleaved image through an interpolating process based on one or morepixels adjacent to each other in the horizontal direction in thecombined images.
 16. A reproducing method of a stereoscopic image, withparallaxes being given in a horizontal direction but not given in avertical direction, the stereoscopic display device including: a displayunit that has a display face on which a parallax interleaved image forstereoscopic display is displayed, with pixels being arranged with afirst horizontal pitch in the horizontal direction; and a parallaxbarrier that has linear optical apertures disposed to face the displayface and arranged with a second horizontal pitch in the horizontaldirection, the optical apertures inclined from the vertical direction,the second horizontal pitch being equal to an integer multiple (n) ofthe first horizontal pitch, the parallax barrier directing light raysemitted from pixels at horizontal intervals of n pixels as parallellight rays toward the viewing zone, the method comprising: preparing aparallax component image data representing n or more parallax componentimages, each having accumulated pixels that cause the pixels to generatethe parallel light rays in the same parallax direction in the viewingzone, and having different numbers of horizontal pixels; recording anultimate combined image that is formed by combining n combined imageshaving the same numbers of vertical and horizontal pixels, the ncombined images being formed by combining one or more parallax componentimages with parallax directions different from each other by n; anddisplaying a parallax interleaved image on the display face afterconverting the ultimate combined image into the parallax interleavedimage.
 17. The reproducing method for a stereoscopic image as claimed inclaim 16, wherein: each of the combined images is a rectangular havingthe same aspect ratio as the parallax interleaved image; and theconversion into the parallax interleaved image involves generation ofpixel data of the parallax interleaved image through an interpolatingprocess based on one or more pixels adjacent to each other in thehorizontal direction in the combined images.
 18. A computer-executableprogram for recording stereoscopic image data for a stereoscopic displaydevice that displays a stereoscopic image, with parallaxes being givenin a horizontal direction but not given in a vertical direction, theprogram comprising instructions for: preparing a parallax componentimage data representing n or more parallax component images, each havingaccumulated pixels that cause the pixels to generate the parallel lightrays in the same parallax direction in the viewing zone, and havingdifferent numbers of horizontal pixels; and recording n combined imageswith the same numbers of vertical and horizontal pixels, the n combinedimages being formed by combining one or more parallax component imageswith parallax directions different from each other by n.
 19. Acomputer-executable reproducing program for displaying a stereoscopicimage, with parallaxes being given in a horizontal direction but notgiven in a vertical direction, the program comprising instructions for:preparing a parallax component image data representing n or moreparallax component images, each having accumulated pixels that cause thepixels to generate the parallel light rays in the same parallaxdirection in the viewing zone, and having different numbers ofhorizontal pixels; recording n combined images with the same numbers ofvertical and horizontal pixels, the n combined images being formed bycombining one or more parallax component images with parallax directionsdifferent from each other by n; and displaying a parallax interleavedimage on a display unit after converting the n combined images into theparallax interleaved image.
 20. A computer-executable reproducingprogram for displaying a stereoscopic image, with parallaxes being givenin a horizontal direction but not given in a vertical direction, theprogram comprising instructions for: preparing a parallax componentimage data representing n or more parallax component images, each havingaccumulated pixels that cause the pixels to generate the parallel lightrays in the same parallax direction in the viewing zone, and havingdifferent numbers of horizontal pixels; recording an ultimate combinedimage that is formed by combining n combined images having the samenumbers of vertical and horizontal pixels, the n combined images beingformed by combining one or more parallax component images with parallaxdirections different from each other by n; and displaying a parallaxinterleaved image on a display unit after converting the ultimatecombined image into the parallax interleaved image.